IN THE CLAIMS : 

Please cancel Claims 137, 139, 154, 156, 161, and 169 without prejudice to or 
disclaimer of the subject matter contained therein. 

Please amend Claims 122-131, 133, 138, 141-146, 147-150, 152-153, 155, 159- 
160, 162-163, 165, 167-168, and 170, and add new Claims 171-174, as follows. 



1-121. (Cancelled) 



122. (Currently Amended) An apparatus for generating annotation data for us e in 
annotating a data file comprising audio data, the apparatus comprising: 

an automatic speech recognizer operable to generate phoneme data for audio data 

in the data file; 

a word decoder operable to process the phoneme data generated by the automatic 
speech recognizer to identify words within the phoneme data generated by the automatic speech 
r ecognizer; and data; 

an annotation generator operable to generate annotation data for the data file by 
combining the generated phoneme data and the words identified by said word decode r, the 
annotation data being for use in subsequent retrieval of the data file; 

an associator operable to associate the generated annotation data with the data file; 

and 
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a storage device operable to store the annotation data in a database to allow the 
associated data file to be retrieved by searching for words and phonemes within the stored 
annotation data. 



123. (Currently Amended) An apparatus for generating annotation data for use in 
annotating a data file, the apparatus comprising: 

an input operable to receive an input voice signal; 

a speech recognizer operable (i) to convert the input voice signal into phoneme 
data and wo r ds: and data and (ii) to process the phoneme data to identify words within the 
phoneme data: 

an annotation generator operable to generate annotation data for the data file by 
combining the phoneme data and the words obtained by said speech recognizer, the annotation 
data being for use in subsequent retrieval of the data file; 

an associator operable to associate the generated annotation data within the data 

file; and 

a storage device operable to store the annotation data in a database to allow the 
associated data file to be retrieved by searching for words and phonemes within the stored 
annotation data. 



124. (Currently Amended) An apparatus for generating annotation data for use in 
annotating a data file, the apparatus comprising: 

an input operable to receive image data representative of text; 
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a character recognizer operable to convert said the image data into text data; 

a converter operable to convert words in the text data into phoneme data; and 

an annotation generator operable to generate annotation data for the data file by 
combining the phoneme data and words in the text data , the annotation data being for use in 
subsequent retrieval of the data file; 

an associator operable to associate the generated annotation data with the data file; 

and 

a storage device operable to store the annotation data in a database to allow the 
associated data file to be retrieved by searching for words and phonemes within the stored 
annotation data . 

125. (Currently Amended) An apparatus according to claim 122, wherein said the 
annotation data defines a phoneme and word lattice and wherein said annotation generator 
comprises: 

(i) a first generator operable to generate data defining a plurality of nodes within 
the lattice and a plurality of links connecting the nodes within the lattice; 

(ii) a second generator operable to generate data associating a plurality of 
phonemes of the phoneme data with a respective plurality of links; and 

(iii) a third generator operable to generate data associating at least one of the 
words with at least one of said the links. 
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126. (Currently Amended) An apparatus according to claim 125, wherein said 
annotation generator is operable to generate said the data defining said the phoneme and word 
lattice in blocks of said the nodes. 

127. (Currently Amended) An apparatus according to claim 125, wherein said 
annotation generator is operable to generate data defining time stamp information for each of said 
the nodes. 

128. (Currently Amended) An apparatus according to claim 127, wherein said 
annotation generator is operable to generate said the phoneme and word lattice data in blocks of 
equal time duration. 

129. (Currently Amended) An apparatus according to claim 126, wherein said 
annotation generator is operable to generate data which defines each block's location within a the 
database. 

130. (Currently Amended) An apparatus according to claim 127, wherein said 
data file includes a time sequential signal, and wherein said annotation generator is operable to 
generate time stamp data which is time synchronized with said the time sequential signal. 

131. (Currently Amended) An apparatus according to claim 122, wherein said 
audio the data |Ue includes audio data which defines the speech of a plurality of speakers, and 
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wherein said annotation generator is operable to generate data which defines separate phoneme 
and word annotation data for the speech of each speaker. 

132. (Previously Presented) An apparatus according to claim 122, wherein said 
speech recognizer is operable to generate data defining a weighting for the phonemes in the 
phoneme data. 

133. (Currently Amended) An apparatus according to claim 122, wherein said 
word decoder is operable to generate data defining a weighting for the words identified within 
said the phoneme data. 

134. (Previously Presented) An apparatus according to claim 125, wherein said 
first generator is operable to define at least one node which is connected to a plurality of other 
nodes by a plurality of links. 

135. (Previously Presented) An apparatus according to claim 123, wherein said 
speech recognizer is operable to generate data defining a weighting for the phonemes in the 
phoneme data. 

136. (Previously Presented) An apparatus according to claim 135, wherein said 
speech recognizer is operable to generate data defining a weighting for the words. 
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137. (Cancelled) 

138. (Currently Amended) An apparatus according to claim 124, wherein said 
converter comprises an automatic phonetic transcription unit which generates said the phoneme 
data from words within the text data output by said character recognizer. 

139. (Cancelled) 

140. (Previously Presented) An apparatus according to claim 124, wherein said 
input comprises a document scanner or a facsimile machine. 

141. (Currently Amended) A method of generating annotation data for use in 
annotating a data file comprising audio data, the method comprising the steps of: 

using an automatic speech recognizer to generate phoneme data for audio data in 

the data file; 

using a word decoder to process the phoneme data generated by the automatic 
speech recognizer to identify words within the phoneme data gene r at e d by the automatic speech 
recognize r : and data; 

generating annotation data for the data file by combining the generated phoneme 
data and the words identified by said the word decoder , the annotation data being for use in 
subsequent retrieval of the data file; 

associating the generated annotation data with the data file; and 
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storing the annotation data in a database to allow the associated data file to be 
retrieved by searching for words and phonemes within the stored annotation data , 

142. (Currently Amended) A method of generating annotation data for use in 
annotating a data file, the method comprising the steps of: 

receiving an input voice signal; 

processing the input voice signal using a speech recognizer to generate phoneme 
data and word data for the input voice signal; and 

processing the generated phoneme data to identify words within the phoneme 

data; 

generating annotation data by combining the phoneme data and the word data 
generated for the input voice signa l the annotation data being for use in subsequent retrieval of 
the data file; 

associating the generated annotation data with the data file; and 
storing the annotation data in a database to allow the associated data file to be y 
retrieved by searching for words and phonemes within the stored annotation data . 

143. (Currently Amended) A method of generating annotation data for use in 
annotating a data file, the method comprising the steps of: 

receiving image data representative of text; 

converting said the image data into text data using a character recognizer; 
converting words in the text data into phoneme data; and 
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generating annotation data for the data file by combining the phoneme data and 
words within the text data , the annotation data beine for use in subseq uent retrieval of the data 
file; 

associating the generated annotation data with the data file; and 
storing the annotation data in a database to allow the associated data file to be 
retrieved bv searching for words and phonemes within the stored annotation data . 

144. (Currently Amended) A method according to claim 141, wherein said the 
annotation data defines a phoneme and word lattice and wherein said generating step comprises 
the steps of: 

(i) generating data defining a plurality of nodes within the lattice and a plurality of 
links connecting the nodes within the lattice; 

(ii) generating data associating a plurality of phonemes of the phoneme data with 
a respective plurality of links; and 

(iii) generating data associating at least one of the words with at least one of said 

the links. 

145. (Currently Amended) A method according to claim 144, wherein said 
generating step generates data defining time stamp information for each of said the nodes. 

146. (Currently Amended) A method according to claim 145, wherein said 
generating step generates data which defines each a block's location within a database. 
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147. (Currently Amended) A method according to claim 145, wherein said the 
data file includes a time sequential signal, and wherein said generating step generates time stamp 
data which is time synchronized with said the time sequential signal. 

148. (Currently Amended) A method according to claim 141, wherein said the 
audio data includes audio data which defines the speech of a plurality of speakers, and wherein 
said generating step generates data which defines separate phoneme and word annotation data for 
the speech of each speaker. 

149. (Currently Amended) A method according to claim 141 claim 144 . wherein 
said the speech recognizer generates data defining a weighting for the phonemes associated with 
said the links. 

150. (Currently Amended) A method according to claim 141 claim 144 , wherein 
said the word decoder generates data defining a weighting for the words associated with said the 
links. 

151. (Previously Presented) A method according to claim 144, wherein said step 
of defining a plurality of nodes and a plurality of links defines at least one node which is 
connected to a plurality of other nodes by a plurality of links. 
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152. (Currently Amended) A method according to claim 142, wherein said the 
speech recognizer generates data defining a weighting for the phon e mes associated with said 
hnks generated phoneme data . 

153. (Currently Amended) A method according to claim 142, wherein said speech 
r ecognize r step of processing the generated phoneme data generates data defining a weighting for 
the words associated with said links identified within the phoneme data . 

154. (Cancelled) 

155. (Currently Amended) A method according to claim 143, wherein said step of 
converting words into phoneme data uses an automatic phonetic transcription unit which 
generates said the phoneme data for words within the text data output by said the character 
recognizer. 

156. (Cancelled) 

157. (Previously Presented) A method according to claim 143, wherein said 
receiving step uses a document scanner or a facsimile machine. 
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158. (Previously Presented) An apparatus according to claim 122, wherein said 
annotation generator is operable to generate annotation data that defines a phoneme and word 
lattice arranged in a time-ordered sequence of blocks. 

1 59. (Currently Amended) An apparatus according to claim 158, wherein said the 
phoneme and word lattice is associated with a time-sequential signal, and wherein said 
annotation generator is operable to generate sard the phoneme and word lattice so that: 

(i) the time-ordered sequence of blocks of the phoneme and word lattice is time- 
synchronized with the time-sequential signal; and 

(ii) each block of the phoneme and word lattice includes an associated time index 
identifying a timing of the block within the time-sequential signal. 

160. (Currently Amended) An apparatus according to claim 159, wherein said 
annotation generator is operable to generate node data which defines a plurality of nodes within 
the lattice, each node representing a point in time at which a word and/or phoneme begins or 
ends within the associated time-sequential signal and wherein each node includes a time-offset 
value defining the point in time represented by the node relative to said the time index associated 
with the corresponding block. 



161. (Cancelled) 
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162. (Currently Amended) An apparatus according to claim 122, wherein said 
automatic speech recognizer is operable to generate phoneme data that includes a phoneme 
lattice which identifies a number of different possible phoneme strings which correspond to 
speech within the audio data, and wherein said annotation generator is operable to generate said 
the annotation data by combining the generated phoneme lattice with the words identified by the 
said word decoder. 



163. (Currently Amended) An apparatus according to claim 123 claim 122 , 
wherein said automatic speech recognizer is operable to generate phoneme data that includes a 
phoneme lattice which identifies a number of different possible phoneme strings which 
correspond to speech within the audio data input voice signal and wherein said annotation 
generator is operable to generate said the annotation data by combining the generated phoneme 
lattice with the words identified by the word decoder said speech recognizer . 

164. (Previously Presented) An apparatus according to claim 122, wherein said, 
annotation generator is operable to generate header data relating to the speech recognizer that 
generated the phoneme data for the annotation. 

165. (Currently Amended) An apparatus according to claim 164, when said 
annotation generator is operable to generate header data including data identifying the language 
and the phoneme sets used by the said automatic speech recognizer. 
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166. (Previously Presented) An apparatus according to claim 123, wherein said 
annotation generator is operable to generate header data relating to the speech recognizer that 
generated the phoneme data for the annotation. 

167. (Currently Amended) An apparatus according to claim 166, wherein said 
annotation generator is operable to generate header data including data identifying the language 
and the phoneme sets used by the said automatic speech recognizer. 



168. (Currently Amended) A computer readable medium storing computer 
program code for causing a programmable processing apparatus to become operable to generate 
annotation data for use in annotating a data file comprising audio data, the cod e computer 
readable medium comprising: 

code for causing an automatic speech recognizer to generate phoneme data for 
audio data in the data file; 

code for causing a word decoder to process the phoneme data generated by the 
automatic speech recognizer to identify words within the phoneme data represented by the 
automatic spe e ch recognize r : and data; 

code for generating annotation data for the data file by combining the generated 
phoneme data and the words identified by the word decode r, the annotation data being for use in 
subsequent retrieval of the data file; 

code for associating the generated annotation data with the data file; and 
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code for storing the annotation data in a database to allow the associated data file 
to be retrieved by searching for words and phonemes within the stored annotation data . 

169. (Cancelled) 

1 70. (Currently Amended) A computer readable medium storing computer 
executable instructions for causing a programmable processing apparatus to become operable to 
generate annotation data for use in annotating a data file comprising audio data , the medium 
comprising: 

code for receiving an input voice signal: 

code for causing an automatic speech recognizer to gene r at e convert the input 

voice signal into phoneme data fo r audio data in the data fil e; 

code for processing the phoneme data to identify words within the phoneme data: 
code fo r causing a word decod e r to identify possible words within the phoneme 

data represented by the automatic speech recognizer; and 

code for generating annotation data by combining the generated phoneme data and 

the identified wo r ds id e ntified by said word decoder words, the annotation data being for use in 

subsequent retrieval of the data file: 

code for associating the generated annotation data with the data file: and 

code for storing the annotation data in a database to allow the associated data file 

to be retrieved by searching for words and phonemes within the stored annotation data . 
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171. (New) A method according to claim 144, wherein said generating step 
generates data defining the phoneme and word lattice in blocks of the nodes. 

1 72. (New) A computer readable medium storing computer executable 
instructions for causing a programmable processing apparatus to become operable to generate 
annotation data for annotating a data file, the medium comprising: 

code for receiving image data representative of text; 

code for converting the image data into text data using a character recognizer; 

code for converting words in the text data into phoneme data; 

code for generating annotation data for the data file by combining the phoneme 
data and words within the text data, the annotation data being for use in subsequent retrieval of 
the data file; 

code for associating the generated annotation data with the data file; and 
code for storing the annotation data in a database to allow the associated data file 
to be retrieved by searching for words and phonemes within the stored annotation data. 

1 73. (New) An apparatus for generating annotation data for use in annotating a 
data file comprising audio data, the apparatus comprising: 

an automatic speech recognizer operable to generate phoneme data for audio data 

in the data file; 

a word decoder operable to identify words within the phoneme data generated by 
said automatic speech recognizer; and 
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an annotation generator operable to generate annotation data by combining the 
generated phoneme data and the words identified by said word decoder and operable to generate 
header data relating to the speech recognizer that generated the phoneme data for the annotation, 
which header data includes data identifying the language and phoneme set used by said automatic 
speech recognizer. 

174. (New) An apparatus for generating annotation data for use in annotating a 
data file, the apparatus comprising: 

an input operable to receive an input voice signal; 

a speech recognizer operable to convert the input voice signal into phoneme data 
and words; and 

an annotation generator operable to generate annotation data by combining the 
phoneme data and the words and operable to generate header data relating to the speech 
recognizer that generated the phoneme data for the annotation, which header data includes data 
identifying the language and the phoneme set used by the speech recognizer. 
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